Intelligent test method for dynamically generating test case according to test performance of tested system

ABSTRACT

An intelligent test method for dynamically generating a test case according to test performance of a tested system. According to the method, n simulation training cases are generated, and the performances of tested agents in the n cases are obtained by testing. By constructing a plurality of decision trees, the algorithm can accurately predict the performances of trained agents in different cases, and learn the spatial division of variables that will lead to different results in the simulation tests, so that the cases can be generated more accurately and effectively in the next round of testing. The method is simple and universal, appropriate for virtual simulation training in various scenes, and thereby improves the effectiveness of case generation in intelligent tests.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority to Chinese Patent ApplicationNo. 202111405653.4, filed on Nov. 24, 2021, the content of which isincorporated herein by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates to the field of automatic drivingsimulation training, in particular to an intelligent test method fordynamically generating a test case according to test performance of atested system.

BACKGROUND

Compared with the training in real scenes, simulation is more efficientand low-cost, so it is widely used in the process of training system.Case generation, as an important link, can facilitate training ofvarious situations that the tested system may encounter in thesimulation environment, and simulating unusual special situations toimprove the training efficiency. In order to test the performance of thesystem under test in various scenarios, the existing case generationmethods cover various real situations by generating a large number ofcases, but this method requires higher requirements for computing powerand time cost. How to dynamically generate effective cases moreintelligently according to the existing performance of an automatictraining system is an urgent problem to be solved.

SUMMARY

In view of the shortcomings of the prior art, the present disclosureprovides an intelligent test method for dynamically generating testcases according to the test performance of a tested system. The methodtakes into account the performance of the trained agent in the previoustest cases in the simulation, and dynamically generates the cases thatare more in line with the test requirements according to the testresults.

The purpose of the present disclosure is realized by the followingtechnical solution:

An intelligent test method for dynamically generating test casesaccording to the test performance of a tested system comprises followingsteps:

S1, selecting a test template test_case_template containing N variablesvar.

S2, sampling values in a value range of each variable var, and forming asampling set Q with all of the sampled values.

S3, selecting a value from a sampling set Q corresponding to eachvariable to form a template variable numerical sampling vector g=(s₁,s₂, . . . , s_(N)), s₁∈Q₁, s₂∈Q₂, . . . , s_(N)∈Q_(N), and thengenerating a set G_init from g.

S4, starting a k^(th) round of test, wherein when k=1, the setG_(k)=G_init; traversing each template variable numerical samplingvector in the set G_(k), obtaining a category label(g) corresponding toeach template variable numerical sampling vector g according to the testtemplate when traversing, and obtaining a test performance data setD_(k) corresponding to the set G_(k).

S5, determining whether k is equal to 1. If k=1, training a decisiontree T_(k) whose classification error rate is less than a presetthreshold epsilon_(k) by the test performance data set D_(k), predictingthe label(g) by the decision tree T_(k) according to any g; and puttinga sample with wrong classification into a data set D_(k+1). If k>1,classifying and verifying the samples in the test performance data setD_(k) by using decision trees T₁, T₂, . . . , T_(k−1), and carrying outweighted average on a classification result of each sample for eachdecision tree to calculate an error rate. If the error rate is greaterthan or equal to a preset threshold epsilon_(k−1), training a decisiontree T_(k) by the test performance data set D_(k), predicting thelabel(g) by the decision tree T_(k) according to any g, putting thesamples with wrong classification into the data set D_(k+1) and going toS6. If the error rate is less than the preset threshold epsilon_(k−1),ending testing the test template test_case_template;

S6, finding all areas in which the classification label of g ispredicted to be the first class by T_(k) in a N-dimensional space whereg is located according to all leaf nodes classified as a first class inT_(k), and taking a union of the areas, wherein the union is denoted asR_(k);

S7, sampling a batch of new vectors g_new in the N-dimensional spacewhere g is located, keeping a vector distribution density positivelycorrelated with a distance between g_new and a boundary of R_(k);forming a set G_new by g_new, letting k=k+1, if k is less than athreshold of preset rounds threshold, letting G_(k)=G_new, going to S4,and starting a next round of testing; otherwise, ending testing the testtemplate test_case_template.

Further, in S4, in order to obtain the data set D_(k) corresponding tothe set G_(k), first, a copy of the test template test_case_template isgenerated, the var_(i) in the copy is replaced by s_(i) in g, and thereplaced copy of the test template is defined as a test case and denotedas H(g); the tested system is tested with H(g) to obtain a test scorescore(g), and then a two-class category label(g) for the test case H(g)is provided according to the score(g), and finally, a binary group (g,label(g)) is put into the test performance data set D_(k).

Further, the step of keeping a vector distribution density positivelycorrelated with a distance between g_new and a boundary of the area R inS7 is specifically that the vector in G_new satisfying the followingconditions:

(1) The closer to the boundary of R_(k) is, the denser the vectordistribution is.

(2) A certain distance exists between a newly sampled vector and atested vector.

Further, the distance between the newly sampled vector g_new and thetested vector g_old satisfies:

An absolute value of projection of g_new−g_old in the j dimension isgreater than or equal to d_(jk), where g_old is an arbitrary vectorbelonging to the union of G₁, G₂, . . . , G_(k), and d_(jk) is thethreshold of the distance between g_new and g_old in the j dimension inthe k^(th) round.

Further, the category label(g) is LOW or NORMAL, the category label(g)is LOW when the test score(g) is less than a preset score threshold, andthe category label(g) is NORMAL when the test score(g) is greater thanor equal to the preset score threshold.

Further, d_(jk) satisfies the following condition:

d _(jk)=alpha*d _(jk-1)

where alpha is a given normal number, 0<alpha<1.

The present disclosure has the following beneficial effects:

In the simulation process, the present disclosure makes full use of theresults of previous test cases, and by constructing a plurality ofdecision trees, the algorithm can accurately predict the performances oftrained agents in different cases, and learn the spatial division ofvariables that will lead to different results in the simulation test.According to this spatial division, the case distributioncharacteristics of the next stage are set to improve the effectivenessof case generation and training efficiency. In addition, the wholeprocess can be iterated online in real time, and the update of thedecision tree is completed in parallel with the case test. At this time,this method can significantly improve the efficiency of simulationtraining, and it is universal and suitable for simulation case testingin various scenarios, and excellent performance can be obtained in manyapplication scenarios that need virtual simulation training.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a flow chart of an intelligent test method for dynamicallygenerating test cases according to the test performance of a testedsystem.

DESCRIPTION OF EMBODIMENTS

The purpose and effect of the present disclosure will become moreapparent from description of the present disclosure in detail accordingto the following drawings and preferred embodiments. It should beunderstood that the specific embodiments described here are only forexplaining the present disclosure, but not for limiting the presentdisclosure.

The method of the present disclosure as shown in FIG. 1 is applied tothe simulation test process of autonomous driving virtual reality, so asto realize the intelligent test of dynamically generating casesaccording to the test performance of the tested system. In such ascenario, this embodiment specifically includes the following steps:

S1, among the cases that are prone to accidents in automatic driving, atest template case test_case_template was selected for the testedautomatic driving agent vehicle to overtake the preceding vehicle. Thetest template contained six variables Var={A_x, A_y, A_v, B_x, B_y,B_v}, where (A_x, A_y) is the current coordinate of the tested agentvehicle with a unit of m; A_v is the current speed of the measuredvehicle with a unit of km/h; (B_x, B_y) is the coordinate unit of thepreceding vehicle of a unit of m; B_v is the speed of the precedingvehicle with a unit of km/h.

Each variable has a definite value range: A_x belongs to [2,4], A_ybelongs to [0,3], B_x belongs to [1,5], B_y belongs to [4,10], A_vbelongs to [30,50] and B_v belongs to [30,60].

S2, values were sampled for 100 times in the value range for eachvariable var, variables were evenly distributed in the value rangethereof during sampling, and a sampling set Q is formed with all thesampled values of each variable var.

S3, a value was selected from a sampling set Q corresponding to eachvariable; a vector g was obtained after each time of sampling, and thena set G_init={g₁, g₂, . . . g₁₀₀} was generated from g, where g_(i) iscalled a template variable numerical sampling vector, and the value ofthe j^(th) dimension of g_(i) corresponds to the numerical sampling ofthe j^(th) variable.

S4, the k^(th) round of test was started. When k=1, the setG_(k)=G_init; d_(k) is the threshold value of the distance between g andg_old in the k^(th) round, d₁=0.1. When k>1, d_(k)=0.9d_(k−1), and thepreset threshold of the error rate epsilon=0.1.

Each template variable numerical sampling vector in the set G_(k) wastraversed, and the category label label(g) corresponding to eachtemplate variable numerical sampling vector g was obtained according tothe test template in the traversal process, and the test performancedata set D_(k) corresponding to the set G_(k) was obtained. The specificoperation was as follows:

(1) A copy of test_case_template of autopilot was generated, var_(i) inthe copy was replaced with s_(i) in G. The test template copy afterreplacement is called as a test case and denoted as H(g).

(2) The tested system was tested with H(g) to get a test score score(g),and then a two-class category label(g) was provided for the test caseH(g) according to the score(g). In an embodiment, the category label(g)was LOW or NORMAL. When the test score(g) was less than the preset scorethreshold, the category label(g) was LOW, and when the test score(g) wasgreater than or equal to the preset score threshold, the categorylabel(g) was NORMAL.

(3) The binary group (g, label(g)) was put into the test performancedata set D_(k).

In the embodiment of automatic driving, the i^(th) dimension numericalsampling g_(i) ^(j) in g_(i) was used to replace the i^(th) templatevariable var_(i) in the test template copy, where i=1, 2, . . . , N. Thetest template copy after replacement is called an automatic driving testcase and denoted as H(g_(i)), where H represents the mappingrelationship from the template variable numerical sampling vector to thegenerated test case. The system under test was tested by the automaticdriving test case H(g_(i)), and the test score of the agent unmannedvehicle was score(g_(i)). Then, a test result label label(g_(i)) was setfor the test case H(g) according to score(g_(i)). The rules for settingthe classification label were as follows: if score(g_(i)) was less than60, then label(g_(i)) was set to low performance LOW, and ifscore(g_(i)) was in a range of [60,100], then label(g_(i)) was set asnormal performance NORMAL, and the binary group (g_(i), label(g_(i)))was put into the test performance data set D_(k).

S5, whether k is equal to 1 was determined. If k=1, D_(k) was used totrain a decision tree T_(k) whose classification error rate is less thanthe preset threshold epsilon_(k), the label(g) was predicted by thedecision tree T_(k) according to any g, and sample with wrongclassification was put into the data set D_(k+1). If k>1, decision treesT₁, T₂, . . . , T_(k−1) were used to classify and verify the samples inD_(k), and weighted average was carried out on a classification resultof each sample for each decision tree to calculate an error rate. If theerror rate was greater than or equal to a preset thresholdepsilon_(k−1), a decision tree T_(k) would be trained by D_(k), thelabel(g) would be predicted by T_(k) according to any g, the sampleswith wrong classification would be put into the data set D_(k+1) andthen S6 would be executed. If the classification error rate was lessthan the preset threshold epsilon_(k−1), the test of the test templatetest_case_templat would be ended.

S6, according to all leaf nodes classified as a first class in T_(k),all areas in which the classification label of g was predicted to be thefirst class by T_(k) are found in the N-dimensional space where g waslocated, and a union of the areas was taken. The union was denoted asR_(k).

In this embodiment of automatic driving, all the leaf nodes classifiedas LOW in Tk are found in the N-dimensional space where G is located,and all the areas predicted by Tk as LOW by the category label of G aremerged and marked as Rk. The category label of any vector outside Rk isNORMAL.

S7, a batch of new vectors g_new was sampled in the N-dimensional spacewhere g is located, and a vector distribution density was keptpositively correlated with a distance between g_new and a boundary ofR_(k); g_new formed a set G_new. Let k=k+1, and if k was less than athreshold of preset rounds, then G_(k)=G_new, S4 would be executed, anda next round of testing would be started; otherwise, the test of thetest template test_case_template would be ended.

The vectors in the set G_new satisfied the following conditions:

(1) In order to make the new test cases approximate to the boundary asmuch as possible, the closer the R_(k) boundary was, the denser thevector distribution was.

(2) In order to reduce the repeated testing of similar cases, there wasa certain distance between the newly sampled vector and the testedvector, and the following conditions were met:

The absolute projection value of g−g_old in the j^(th) dimension wasgreater than or equal to d_(jk), where g_old is an arbitrary vectorbelonging to the union of G₁, G₂, . . . , G_(k), and d_(jk) is thethreshold of the distance between g and g_old in the j^(th) dimension inthe k^(th) round.

As the number of iteration rounds increased, the accumulated vectorsgradually increased. In order to reduce the amount of data calculation,d_(jk) was set to be a gradually decreasing value, namely:

d _(jk)=alpha*d _(jk−1)

where alpha is a given normal number, 0<alpha<1.

In an embodiment of automatic driving, alpha is 0.9.

The intelligent test method of the present disclosure is applied to theautomatic driving scene, which can influence the algorithm logic ofautomatic driving case selection, so that the case that is more likelyto be tested as the junction of low and normal is selected, and thetraining efficiency of automatic driving in the virtual training groundis increased.

It can be understood by those skilled in the art that the above examplesare only preferred examples of the present disclosure, and are notintended to limit the present disclosure. Although the presentdisclosure has been described in detail with reference to the aboveexamples, it is still possible for those skilled in the art to modifythe technical solutions described in the above examples or equivalentlyreplace some of the technical features. Modifications, equivalents andsubstitutions made within the spirit and principle of the presentdisclosure shall be included in the scope of protection of the presentdisclosure.

What is claimed is:
 1. An intelligent test method for dynamicallygenerating a test case based on test performance of a tested system,comprising following steps: S1, selecting a test templatetest_case_template containing N variables var; S2, sampling values in avalue range of each variable var and forming a sampling set Q with allof the sampled values; S3, selecting a value from a sampling set Qcorresponding to each variable to form a template variable numericalsampling vector g=(s₁, s₂, . . . , s_(N)), s₁∈Q₁, s₂∈Q₂, . . . ,s_(N)∈Q_(N), and then generating a set G_init from g; S4, starting ak^(th) round of test, wherein when k=1, the set G_(k)=G_init; traversingeach template variable numerical sampling vector in the set G_(k),obtaining a category label(g) corresponding to each template variablenumerical sampling vector g according to the test template whentraversing, and obtaining a test performance data set D_(k)corresponding to the set G_(k); S5, determining whether k is equal to 1,if k=1, training a decision tree T_(k) whose error rate is less than apreset threshold epsilon_(k) by the test performance data set D_(k),predicting the label(g) by the decision tree T_(k) according to any g,and putting a sample with wrong classification into a data set D_(k+1);if k>1, classifying and verifying the samples in test performance dataset D_(k) by decision trees T₁, T₂, . . . , T_(k−1), and carrying outweighted average on a classification result of each sample for eachdecision tree to calculate an error rate, if the error rate is greaterthan or equal to a preset threshold epsilon_(k−1), training a by testperformance data set D_(k), predicting the label(g) by the decision treeT_(k) according to any g, putting the samples with wrong classificationinto the data set D_(k+1), and going to S6, if the error rate is lessthan the preset threshold epsilon_(k−1), ending testing the testtemplate test_case_template; S6, finding all areas in which theclassification label of g is predicted to be the first class by thedecision tree T_(k) in a N-dimensional space where g is locatedaccording to all leaf nodes classified as a first class in the decisiontree T_(k) and taking a union of the areas, wherein the union is denotedas R_(k); S7, sampling a batch of new vectors g_new in the N-dimensionalspace where g is located, keeping a vector distribution densitypositively correlated with a distance between g_new and a boundary ofR_(k), forming a set G_new by the vectors g_new; letting k=k+1, if k isless than a threshold of preset rounds of testing, letting G_(k)=G_new,going to S4, and starting a next round of testing, and if k is greaterand equal to the threshold of preset rounds of testing, ending testingthe test template test_case_template.
 2. The intelligent test method fordynamically generating a test case according to test performance of atested system according to claim 1, wherein step S4 further comprises inorder to obtain the data set D_(k) corresponding to the set G_(k), firstgenerating a copy of the test template test_case_template, replacing thevariable var_(i) in the copy by s_(i) in g, wherein the replaced copy ofthe test template is defined as a test case and denoted as H(g); testingthe tested system with H(g) to obtain a test score(g), and providing atwo-class category label(g) for the test case H(g) according to the testscore(g); and finally, putting a binary group (g, label(g)) into thetest performance data set D_(k).
 3. The intelligent test method fordynamically generating a test case according to test performance of atested system according to claim 1, wherein said keeping a vectordistribution density positively correlated with a distance between g_newand a boundary of the area R in S7 comprises the vector in G_newsatisfying following conditions: (1) the closer to the boundary of R_(k)is, the denser the vector distribution is; and (2) a certain distanceexists between a newly sampled vector and a tested vector.
 4. Theintelligent test method for dynamically generating a test case accordingto test performance of a tested system according to claim 3, wherein thedistance between the newly sampled vector g_new and the tested vectorg_old satisfies: an absolute value of projection of g_new-g_old in a jdimension is greater than or equal to d_(jk), where g_old is anarbitrary vector belonging to a union of G₁, G₂, . . . , G_(k), andd_(jk) is a threshold of a distance between g_new and g_old in the jdimension in a k^(th) round.
 5. The intelligent test method fordynamically generating a test case according to test performance of atested system according to claim 1, wherein the category label(g) is LOWor NORMAL, the category label(g) being LOW when the test score(g) isless than a preset score threshold, and the category label(g) is NORMALwhen the test score(g) is greater than or equal to the preset scorethreshold.
 6. The intelligent test method for dynamically generating atest case according to test performance of a tested system according toclaim 4, wherein dik satisfies following condition:d _(jk)=alpha*d _(jk−1) where alpha is a given normal number, 0<alpha<1.